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ABSTRACT:  Accurately  describing  simulation  characteristics  is  essential  for  their  interoperability  and  compos  ability. 
The  internal  processes  and  results  produced  must  be  rigorously  characterized  and  valid  in  the  context  of  their 
application.  Only  if  this  is  possible  in  an  effective  and  economic  manner  can  simulations  be  extensively  reused,  through 
either  composability  or  tailoring.  To  describe  simulations  accurately  and  with  confidence,  and  to  ensure  that  the 
simulation  is  valid  in  the  context  of  the  application,  an  understanding  of  the  model  semantics  and  semantic 
interdependencies  is  required.  This  paper  describes  an  initial  effort  to  develop  a  set  of  systematic  descriptors  that 
capture  the  characteristics  and  interdependencies  of  a  simulation.  We  describe  a  framework  for  the  identification  and 
organization  of  semantic  descriptors  and  provide  some  examples.  We  discuss  a  process  for  identifying  associated 
metrics  and  present  an  experimental  design  for  developing  their  scales.  The  paper  concludes  by  presenting  some 
experimental  results  and  by  discussing  directions  for  continuing  and  building  upon  this  work. 


1.  Introduction 

Description  of  models  and  simulations  exist  at  two  basic 
levels;  functional  and  syntactic.  The  functional 
description  provides  a  general  understanding  of  the 
purpose  of  the  model  such  as  radar  performance  or  entity 
level  battlespace  representation.  The  syntactic  description 
provides  an  understanding  of  the  inputs  and  outputs  of  the 
model  or  simulation.  It  is  the  syntactic  level  that  is 
currently  most  often  used  to  address  issues  of 
interoperability.  If  the  two  models  can  exchange  data  in  a 
prescribed  format,  they  are  considered  interoperable. 

These  simulation  descriptions  are,  however,  insufficient 
in  some  very  key  areas.  They  offer  little  or  no  insight  into 
the  constraints  and  assumptions  implicit  in  the  models.  It 


is  therefore  difficult,  if  not  impossible  to  determine  if  the 
results  from  a  model  would  be  valid  in  a  specific  context. 
It  is  also  difficult  to  determine  if  simulations  that  are 
federated  will  produce  coherent  or  consistent  results.  Lack 
of  such  semantic  interoperability  can  lead  to  outcomes 
that  are  invalid  or  inaccurate,  perhaps  in  subtle  ways  that 
are  not  immediately  obvious. 

The  establishment  of  well  structured  semantic  descriptors, 
therefore,  could  play  a  significant  role  in  several  areas  of 
the  design  and  use  of  simulations.  The  information  they 
provide  would  be  essential  to  the  realization  of 
composable  simulations.  This  information  could  also  be 
used  to  assess  the  validity  of  a  simulation  federation  and 
define  the  context  under  which  that  validity  would  hold. 
That  is,  they  could  help  define  the  limits  of  the  utility  of  a 
federation. 
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The  association  of  semantic  descriptors  with  context 
validity  also  suggests  a  role  for  them  in  requirements 
definition.  A  framework  for  the  organization  of 
descriptors  that  address  the  capabilities  of  models  could 
also  provide  an  efficient  means  to  identify  and  organize 
simulation  requirements  statements.  If  the  same  language 
was  used  for  both  simulation  requirements  and  a 
description  of  model  capability,  the  groundwork  would  be 
in  place  for  development  of  automated  user-defined 
composable  simulations. 

2.  Goal  and  objectives 

The  goal  of  this  research  is  to  provide  a  foundation  for 
developing  semantic  descriptions  of  models  and 
simulations.  To  provide  this  foundation,  this  research  has 
the  following  specific  objectives: 

1 .  Recommend  an  infrastructure  for  the 
organization  and  definition  of  semantic 
descriptors, 

2.  Develop  an  initial  set  of  semantic  descriptors  for 
models  and  simulations, 

3.  Propose  an  approach  for  the  assignment  of 
metrics  to  be  associated  with  each  semantic 
descriptor, 

4.  Identify  procedures  for  evaluating  the  validity 
and  applicability  of  candidate  descriptors. 

3.  Semantic  descriptor  definition  and 
framework 

3.1  Working  definition 

As  a  starting  point,  a  working  definition  of  a  semantic 
descriptor  is  needed.  In  this  definition,  it  is  important  that 
both  the  functionality  and  limitations  of  semantic 
descriptors  are  articulated.  That  is,  a  definition  of  what  a 
semantic  descriptor  is  and  is  not  needed  to  be  developed. 


Fundamentally,  the  semantic  descriptors  should  capture 
the  assumptions  and  abstractions  relevant  to  the  model. 
They  ought  to  convey  a  sense  of  quality  and  associated 
operational  range.  Some  descriptors  will  reflect  the 
direction,  dependency,  and  dimension  of  information 
flow.  These  will  likely  explicitly  address  inputs,  outputs, 
and  connectivity.  As  a  result,  some  semantic  descriptors 
will  need  to  incorporate  syntactic  data.  For  example,  if  a 
descriptor  addresses  accuracy,  it  will  need  to  include 
units,  such  as  ±0.1  dB  re  pPa. 

In  the  context  of  model  interoperability  and  composable 
simulations,  an  analogy  to  the  direction,  dependency,  and 
dimension  of  information  flow  is  represented  by  the 
semantic  descriptors  aids  in  visualization.  Consider 
“Lego-type”  blocks  in  which  the  pegs  and  holes  are 
neither  of  uniform  size  nor  in  uniform  locations.  The  pegs 
and  holes  are  analogous  to  the  direction,  dependency,  and 
dimension  of  information  flow  between  models.  If  the 
size  and  location  of  the  pegs  on  one  block  match  the  size 
and  location  of  the  holes  of  another  block,  the  blocks  can 
be  joined.  There  are  also  conditions  under  which  a  precise 
match  of  peg  and  hole  size  and  location  are  not  necessary 
for  the  blocks  to  be  joined.  Blocks  may  be  flush  if  the  peg 
is  smaller  than  the  hole,  or  no  peg  is  present.  This  results 
in  the  blocks  not  being  held  together  as  tightly  as  when 
holes  and  pegs  precisely  match,  but  this  may  be  good 
enough  for  the  structure  being  built.  This  analogous  to 
two  models  not  being  a  perfect  fit  for  each  other,  but  their 
interoperability  is  sufficient  for  the  specific  purpose  at 
hand. 

Misalignment  of  pegs  and  holes  is  analogous  to  a 
mismatch  of  syntax  between  the  models.  One  model 
requires  one  an  input  of  X,  but  the  other  model  has  an 
output  of  Y.  There  can  be  cases  where  the  syntax  agrees, 
however,  and  yet  the  models  do  not  interoperate  properly. 
This  would  be  represented  in  the  Lego  analogy  as  the  peg 
being  too  big  for  the  hole.  Figure  1  represents  the  when 
models  can  work  together  or  not  work  together  using  this 
Lego  analogy. 


Syntactic  agreement  Syntactic  disagreement 

Semantic  disagreement 

Figure  1.  Lego  representation  of  descriptors 


The  concept  that  models  may  not  be  semantically  equal, 
but  sufficiently  close  to  jointly  satisfy  some  specific 
application  again  highlights  the  tie  between  the  semantic 
descriptors  and  requirements.  A  semantic  descriptor  will 
have  a  specific  metric  value  of  a  for  a  specific  model 
implementation.  Simulation  requirements  can  then  be 
stated  in  terms  of  the  metric  values  needed  to  provide 
types  and  level  of  results  required.  Requirements 
statements  such  as  “I  need  a  simulation  with  a  minimum 
value  of  X  for  descriptor  A  and  Y  for  descriptor  B”  can  be 
formed  to  assist  in  the  automated  composition  of 
simulations  to  meet  the  current  need. 

Semantic  descriptors  are  not  completely  descriptive  of 
simulations,  however.  Some  information  critical  to 
determining  model  interoperability  is  not  currently 
included  as  part  of  a  semantic  descriptor.  Examples  of 
such  information  would  be  a  model’s  security 
classification  and  its  execution  speed  relative  to  wall 
clock  time.  Such  aspects  do  not  generally  address  the 
assumptions  and  abstractions  made  in  model  development 
but  rather  deal  with  implementation  decisions.  As  the 
definition  of  semantic  descriptors  matures,  it  may  be 
expanded  to  include  such  additional  factors. 

3.2  Organizational  framework 

To  avoid  being  based  on  any  specific  instantiation, 
semantic  descriptors  should  be  based  on  conceptual 
models.  There  are  many  options  as  to  the  type  of 
conceptual  models  to  be  used  as  the  basis  for  the  semantic 


descriptors.  One  is  to  use  entity  types  as  a  basis.  Another 
is  to  use  a  model  of  functionality. 

Basing  semantic  descriptors  on  entities  has  some 
drawbacks.  This  results  in  conceptual  models  that  are  not 
based  on  interactions  and  often  have  more  of  a  database 
appearance.  The  need  for  many  conceptual  models  is 
highly  probable.  Exacerbating  this  is  the  need  for  many 
redundant  descriptors  across  the  collection  of  conceptual 
models  since  many  entities  can  have  the  same  or  similar 
functional  roles  and  behaviors. 

A  logical  next  step  is  then  to  investigate  a  framework  for 
semantic  descriptors  that  is  based  on  functional  areas.  The 
premise  is  that  relatively  few  general  functions  can  be 
used  to  completely  describe  any  simulation  scenario.  This 
allows  for  the  use  of  a  small  number  of  types  of 
conceptual  models  and  eases  the  configuration 
management  of  the  semantic  descriptor  system. 
Additionally,  basic  functions  apply  to  all  levels  of 
resolution  that  may  be  modeled  so  different  domains  are 
not  required  to  address  components  of  objects,  objects, 
and  collections  of  objects. 

Drawing  on  some  of  the  principles  of  the  Object 
Management  Group’s  Model  Driven  Architecture 
approach  to  software  design,  these  functional  areas  should 
be  highly  cohesive  and  loosely  coupled.  By  highly 
coherent  is  meant  that  all  functions  within  a  domain  are 
closely  and  naturally  related.  Such  relationships  could  be, 
for  example,  different  instantiations  of  the  same  function 


or  actions  that  combine  in  a  sequential  or  cyclic  manner  to 
result  in  a  given  function.  The  cohesion  within  a  domain 
lends  itself  to  easy  analysis  by  a  subject  matter  expert.  It 
also  allows  the  domains  to  be  defined  tightly  enough  so 
that  there  is  little  doubt  as  to  what  functions  belong  in 
each  domain.  By  loosely  coupled  is  meant  that  the 
interdependencies  between  domains  should  be  minimal 
and  the  interfaces  can  be  well  defined  in  precise  contracts. 

Based  on  this,  the  requirements  for  the  functional  areas 
can  be  established.  The  functional  areas  need  to 
encompass  the  fundamental  actions  that  comprise  the 
(military)  operations  to  be  simulated.  Each  functional  area 
needs  to  be  broad  enough  to  accommodate  many  forms  of 
the  generic  actions.  This  is  desired  so  the  number  of 
functional  areas  can  be  kept  manageably  small.  At  the 
same  time,  a  functional  area  must  be  specific  enough  to 
be  clearly  distinguished  from  other  functional  areas. 

This  still  leaves  the  tasks  of  identifying  the  functional 
areas  to  be  used  as  a  basis  for  semantic  descriptors.  To 
those  familiar  with  military  operations,  a  breakdown  of 
functionality  that  may  at  first  seem  natural  to  modeling 
military  operations  is  that  of  warfare  areas.  Upon  closer 
examination,  however,  it  is  soon  discovered  that  this 
choice,  like  that  of  entity-based  conceptual  models  above, 
would  result  in  the  need  for  excessively  redundant 
descriptors  throughout  the  functional  areas.  That  is,  the 
functions  within  a  specific  warfare  area  are  composed  of 
functions  that  are  not  unique  to  that  warfare  area.  Because 
of  this,  a  more  fundamental  list  of  functional  areas  upon 
which  to  base  the  framework  for  semantic  descriptors  is 
proposed.  This  initial  set  of  functional  areas  is  comprised 
of  move,  sense,  engage,  navigate,  command  and  control 
(C2),  communicate,  exist,  and  environment. 

Since  models  address  the  world  at  various  levels  of 
abstraction,  the  framework  for  semantic  descriptors  must 
also  address  these  levels.  This  suggests  a  hierarchical 
structure  to  the  functional  area  based  framework.  A 
notional  three  level  hierarchy  for  each  functional  domain 
is  also  proposed.  The  functions  at  each  level  are  identified 
as  components  of  conceptual  models  of  the  appropriate 
resolution.  The  three  proposed  levels  of  the  hierarchy 
within  each  domain  are  the  aggregate,  platform,  and 
component  levels.  These  three  levels  have  been 
extensively  used  within  the  simulation  community  for  a 
number  of  years.  They  are  often  called,  for  example, 
theater,  entity,  and  engineering.  This  current  research 
effort  does  not  attempt  to  propose  a  formal  definition  of 
these  levels  but  instead  will  rely  on  the  reader’s  intuition 


as  to  what  these  levels  mean  and  offer  only  these  casual 
observations. 

•  Typically,  discussions  of  aggregate  level 
simulations  involve  theater  or  strategic  operations. 

•  The  term  platform  level  is  applied  to  simulations 
that  represent  individual  combat  entities  such  as 
aircraft  and  armored  vehicles,  but  can  be  large  and 
complex  as  an  aircraft  carrier  or  a  small  as  an 
individual  combatant. 

•  Component  level  simulations  typically  represent 
individual  combat  systems  or  mechanisms  of 
combat  systems  or  entities. 

These  hierarchical  levels  are  used  here  in  much  the  same 
way  as  the  simulation  community  has  for  years,  but  with 
one  critical  difference.  Instead  of  using  them  for  the 
classification  of  simulations,  they  are  used  to  stratify 
functional  domains. 

Within  these  functional  areas,  there  is  a  natural  flow  down 
from  one  level  of  the  hierarchy  to  the  next.  There  is  also, 
in  general,  a  natural  expansion  to  the  number  of  functions 
as  resolution  increases.  This  arises  for  two  reasons.  The 
first  is  the  natural  consequence  of  additional  detail.  The 
second  is  that  there  may  be  multiple  different  breakdowns 
of  detail  needed  to  account  for  the  ways  different  objects 
perform  the  same  function.  Consider,  as  an  example,  the 
entity  level  function  of  “move  (at  a  given  velocity).”  If 
the  platform  is  a  submarine,  the  functional  decomposition 
could  include  functions  such  as  generate  steam  and  rotate 
shaft,  while  for  an  aircraft  they  could  include  ignite  jet 
fuel  and  rotate  turbine.  In  general,  the  functional 
decompositions  should  be  kept  as  generic  and  widely 
applicable  as  possible.  Therefore,  it  could  be  argued  that  a 
preferable  decomposition  to  “move”  might  be  something 
like  convert  energy  form  and  generate  thrust.  Still,  it  is 
probable  that  at  some  level  the  fundamental  differences 
between  the  functions  of  an  aircraft  and  its  parts  and  a 
submarine  and  its  parts  must  be  represented. 

The  definition  of  the  domains  and  example  functions  at 
each  level  of  resolution  are  provided  in  Table  1.  It  is 
recognized  that  additional  functional  areas  will  likely  be 
required  such  as  political  or  socio-economic  environment. 
This  preliminary  set  is  sufficient,  however,  for  the  initial 
presentation  and  discussion  of  the  semantic  descriptor 
framework  needed  for  community  vetting.  Within  the 
hierarchical  based  framework,  the  semantic  descriptors 
associated  with  any  given  functional  domain  provide  an 
understanding  of  the  level  of  resolution  at  which  the 
model  operates. 


Domain 

Definition 

Aggregate 

Platform 

Component 

Move 

Relocation  of  objects  through 
the  battlespace 

Battle  force 
center  of  mass 

Entity  velocity  and 
acceleration  vectors 

Fuel  state 

Rate  of  turn 

Lubricate 

Combust 

Rotate 

Jet  engines 

Ship  engineering 
plant  configuration 

Sense 

Passive  and  active  means  of 
detection,  localization, 
tracking,  and  identifying 
other  objects  in  the  virtual 
battlespace 

Receive  and 
interpret 
intelligence 
reports 

Coverage  of 
organic  sensors 

Performance  of 
organic  sensors 

Sensor  beam  patterns 

Pulse  repetition  rate 
Transmit  power 
Element  level 
signal 

Detailed  signal-to- 
noise  ratio 

Engage 

Any  active  or  passive  means 
of  interaction  with  other 
objects 

Force  on  force 

Detect,  generate  fire 
control  solution,  fire 
weapon; 

Range  gate  pull-off; 
Missile  vs.  aircraft; 

Tank  round  vs.  BRDM 

Seeker  head 
movement;  location 
of  expanding  rod 
fragments  vs. 
aircraft  component 
location 

Navigate 

The  means  of  identifying 
location  in  the  battlespace 
and  determining  the 
movement  required  to  carry 
out  the  assigned  mission 

Respond  to 
commands  to 
move  or  deploy 

Determine  position 

GPS  posit 

Come  to  course  and 
speed 

Left  10  degrees 
rudder;  thrust  vs. 
airspeed 

C2 

Behaviors  and  logic  for 
determining  and  directing 
courses  of  action 

OPORDER 

Attack  by  fire;  assume 
plane  guard  station 

Servo  motor 
commands 

Communicate 

Passing  data,  information,  or 
perception 

Shared  COP 

OTH  Gold 

Data  bits 

Link- 16 

Exist 

Aspects  of  physical  objects 
that  arise  because  of  physical 
existence  and  not  contained 
in  other  functional  areas 

The  center  of 
mass  of  a  battle 
group 

Mass 

Volume 

Composition 

Coefficient  of 
friction,  tensile 
strength 

Environment 

Representations  of  the  natural 
environment 

Air  mass 

Ocean  areas 

SVP  by  region 

Complex  fluid  flow 
on  molecular  level 

Table  1.  Proposed  functional  domains  and  hierarchy  for  semantic  descriptors 


For  any  scheme  used  as  the  foundation  of  describing 
qualities  of  models,  there  will  be  areas  of  uncertainty  in 
its  implementation,  and  the  above  scheme  of  functional 
domains  is  no  exception.  While  one  of  the  major 
objectives  in  the  development  of  the  above  scheme  was  to 
ensure  non-ambiguous  assignment  of  functions  to 
domains,  there  are  still  implementation  decisions  to  be 
made.  The  following  discussion  provides  examples  of  two 
such  cases. 

Consider  the  issue  of  energy  signatures.  Such  signatures 
arise  from  activities  such  as  the  radiation  of  a  radar  to 
sense,  the  noise  of  a  torpedo  launch,  or  the  thermal  trail 
left  by  the  engines  moving  a  jet  aircraft.  The  signature  is 
an  unavoidable  consequence  of  the  functions  and  so 
descriptions  of  the  models  of  signatures  could  logically  be 


placed  in  the  respective  functional  areas.  It  could  also  be 
argued,  however,  that  signatures  are  a  consequence  of  an 
objects  existence  in  a  certain  state  (the  state  of  radiating  a 
radar  or  flying  through  the  air,  for  example),  and  therefore 
all  signature  representations  ought  to  be  contained  under 
the  “exist”  function.  This  would  have  the  potential 
advantage  of  placing  all  semantic  descriptors  associated 
with  signatures  or  operational  state  in  one  functional 
domain.  A  potential  disadvantage  is  that  this  may  force 
tight  coupling  between  semantic  descriptors  in  different 
functional  domains.  For  example,  the  semantic  descriptor 
associated  with  the  resolution  of  a  radar’s  radiation 
pattern  (and  thus  detection  performance)  would  be  closely 
related  to  the  semantic  descriptor  of  its  signature 
description.  It  is  currently  unclear  if  either  option 
provides  a  clear  operational  benefit. 


As  another  implementation  issue,  consider  the  logic 
associated  with  many  functions.  As  a  minimum,  functions 
in  the  domains  of  sense,  engage,  navigate,  command  and 
control,  and  communicate  require  logic  to  determine  such 
things  as  target  classification,  when  to  engage,  course  and 
speed  required  to  reach  a  specific  point  at  a  specific  time, 
when  to  send  a  message,  and  what  information  the 
message  should  contain.  Analogous  to  the  above 
discussion,  such  logic  could  be  embedded  in  the 
appropriate  functional  domain,  or  could  all  reside  in  a 
single  domain;  command  and  control  in  this  instance.  The 
argument  for  putting  all  the  logic  in  one  domain  is  that 
such  logic  falls  under  the  actions  of  command  and  control 
and,  therefore,  should  be  concentrated  in  that  domain.  The 
benefit  of  this  is  the  avoidance  of  some  redundant  or 
nearly  redundant  descriptor  constructs  in  multiple 
domains.  The  disadvantage  is  again  forcing  a  tight 
coupling  between  domains  that  might  otherwise  not  be 
necessary.  Further  study  is  required  to  determine  if  one 
implementation  has  benefits  over  the  other. 

4.  Metrics 

Semantic  descriptors  must  not  only  convey  what 
attributes  a  model  has,  but  must  also  provide  a  sense  as  to 
the  degree  to  which  it  possesses  each  of  those  attributes. 
This  highlights  the  need  for  measurement  or  metric 
associated  with  each  semantic  descriptor.  Like  the 
descriptors  themselves,  the  scale  for  the  metrics  needs  to 
be  model  and  implementation  independent. 

In  the  formulation  of  metrics,  all  relevant  scales  and  value 
types  are  being  considered.  This  includes  nominal, 
ordinal,  interval,  and  ratio.  Each  of  these  types  is 
appropriate  for  different  semantic  descriptors  and  for 
some  descriptors,  multiple  types  might  be  appropriate. 
When  multiple  types  are  applicable,  they  can  be  assigned 
in  an  hierarchical  manner.  The  presence  of  the 
representation  of  an  attribute  (nominal)  could  be  checked 
first.  If  the  attribute  is  represented,  its  quality  on  an 
ordinal  scale  and  interval  scale  could  be  checked  for 
suitability  for  the  intended  application. 

As  an  example  of  a  multi-metric  semantic  descriptor, 
consider  a  descriptor  that  addresses  the  number  of 
organizational  levels  represented  in  command  and  control 
(C2)  simulations.  This  descriptor  is  to  be  evaluated 
relative  to  two  simulation;  one  that  addresses  the 
interactions  between  an  Air  Intercept  Controller  (AIC) 
and  an  aircraft,  and  another  that  considers  the  interactions 
between  the  Battle  Group,  the  carrier’s  air  boss,  the  AIC, 
the  aircraft,  and  the  Radar  Intercept  Officer  (RIO).  For 
both  simulations,  the  initial  nominal  evaluation  of  the 
descriptor  would  be  “yes.”  That  is,  each  simulation 
addresses  multiple  organizational  levels.  Evaluation  of  the 


ordinal  value  would  then  be  required.  For  the  first 
simulation,  this  might  have  a  value  set  like  (max  2,  min 
1).  For  the  second,  it  might  be  (max  4,  min  1), 
representing  the  maximum  and  minimum  number  of 
levels  represented.  Continuing  with  the  assignment  of 
metrics,  the  assignment  of  interval  metric  data  might  yield 
values  like  2/1  &  1,  and  4/1  &  3  respectively  for  these 
simulations,  representing  the  ratio  of  the  maximum  to  the 
minimum  values  and  the  value  most  often  found.  For  a 
simulation  which  for  which  the  nominal  value  was  “no,” 
no  assessment  of  ordinal  or  interval  data  types  would  be 
required  or  appropriate. 

Many  aspects  of  a  model’s  representation  are  naturally 
ratio  data.  Physical  attributes  are  the  most  obvious 
example  of  this.  Representation  of  weight  to  within 
0.001%  accuracy,  or  bearing  resolution  to  within  0.1 
degrees  would  be  examples  of  this. 

An  open  question  being  addressed  through 
experimentation  is  if  appropriate  parametric  ranges  can  be 
associated  with  such  scales.  Such  a  notional  relationship 
is  shown  in  Figure  2.  The  questions  to  be  addressed  are 
not  only  what  the  boundaries  of  the  ranges  are,  but  what  is 
the  significance  or  applicability  of  the  ranges.  It  could  be, 
for  example,  the  models  that  both  fall  within  the  same 
range  could  be  expected  to  provide  satisfactory 
interoperability  for  specified  applications,  but  models  in 
different  ranges  should  not  be  expected  to  provide 
suitable  interoperability  for  those  applications. 
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Figure  2.  Notional  parametric  ranges 


Like  the  association  of  two  different  scale  types  to 
represent  one  semantic  descriptor,  there  will  be  cases 
where  there  will  be  relationships  between  different 


semantic  descriptors.  In  these  cases  it  will  be  necessary  to 
account  for  interdependence,  correlation,  or  even 
dominance  amongst  or  between  descriptors.  Such 
relationships  foreshadow  the  utility  of  multi-variant 
semantic  descriptor  values.  Although  it  is  beyond  this 
initial  effort  to  characterize  and  quantify  these  values,  the 
structure  being  developed  allows  for  their  addition. 

5.  Experiments 

5.1  Purpose  and  design  of  the  experiments 

A  key  component  of  this  research  is  experimentation.  The 
data  from  the  experiments  will  support  two  critical 
analyses  relative  to  the  semantic  descriptors.  First,  the 
experiments  will  be  used  to  test  the  relevance  of  candidate 
descriptors.  Secondly,  the  experiments  will  provide  data 
to  support  the  determination  of  the  applicability  and 
validity  of  metrics  associated  with  each  descriptor.  Part  of 
this  second  analysis  will  be,  as  discussed  above, 
investigation  into  the  feasibility  of  assigning  parametric 
ranges  to  interval  or  ratio  valued  descriptors. 

To  support  this  initial  experimentation  on  semantic 
descriptors,  we  constructed  two  simple  model  sets.  The 
use  of  new  models  provided  two  distinct  advantages  over 
the  use  of  existing  models.  First,  the  new  models  were 
constructed  to  allow  for  the  easy  and  highly  controllable 
modification  of  most  of  their  aspects.  This  control  is 
much  more  than  simply  controlling  input  parameters 
associated  with  a  model.  These  models  were  constructed 
such  that  aspects  of  fidelity  and  accuracy  were  easily 
controllable.  The  second  advantage  of  using  new  models 
is  that  we  had  total  insight  into  their  construction  and 
associated  assumptions.  When  an  aspect  of  the  model 
associated  with  a  semantic  descriptor  was  modified,  this 
insight  provided  an  understanding  of  exactly  what  was 
changing  about  the  model  and  provided  confidence  that 
unknown  secondary  effects  were  not  affecting  the 
experiment. 

Testing  the  relevance  of  candidate  descriptors  is 
essentially  a  sensitivity  analysis.  The  model  aspect 
associated  with  a  candidate  descriptor  is  varied.  The 
results  of  repeated  experiments  are  then  analyzed  to  see  if 
that  variation  affected  the  interoperability  of  that  model 
with  the  other  models  in  the  set.  As  an  example  of  how 
this  might  be  done,  consider  a  descriptor  associated  with 
accuracy  of  a  propagation-loss  calculation.  The  model 
could  be  constructed  to  add  a  known  amount  of  white 
noise.  In  this  manner,  its  accuracy  and  the  value  of  the 
associated  semantic  descriptor  could  be  altered  in  the 
controllable  manner  needed  for  repeatable 
experimentation.  An  aspect  of  the  model  associated  with 
one  candidate  semantic  descriptor  is  varied  at  a  time.  If 
varying  the  model  aspect,  and  thus  the  value  of  the 


model’s  candidate  descriptor,  affect  the  model’s 
interoperability  with  other  models,  the  candidate 
descriptor  is  a  relevant  one. 

Assessment  of  applicability  and  validity  of  the  metrics 
associated  with  a  descriptor  requires  the  careful 
association  of  values  with  representation  types  and  levels. 
Once  a  representation  structure  is  developed, 
investigation  into  assignment  of  values  or  parametric 
ranges  of  interval  or  ratio  data  is  done  empirically.  The 
aspect  of  interest  is  systematically  varied.  Break  points  in 
the  degree  or  manner  of  interoperability  with  the  other 
models  of  the  set  are  then  sought.  The  boundaries 
between  ranges  occur  at  these  break  points. 

Two  different  model  sets  were  developed  to  support  the 
testing  of  semantic  descriptors.  One  set  addresses  physics- 
based  models  and  the  other  addresses  command,  control, 
and  communications  (C3)  models.  The  physics-based 
models  are  for  an  anti-submarine  warfare  (ASW) 
scenario.  It  employs  two  submarines  (red  and  blue)  that 
attempt  to  detect  and  engage  each  other.  One  submarine  is 
executing  a  search  pattern  while  the  other  is  executing  a 
transit  through  the  area.  The  basic  approach  to  testing 
interoperability  is  to  assess  deviation  from  an  established 
baseline.  A  number  of  runs  are  made  with  all  aspects  of 
both  submarine  models  identical  with  the  exception  of 
tasking  as  patrolling  or  transiting.  This  establishes  the 
definition  of  “a  level  playing  field”  for  the  scenario. 
Aspects  of  the  models  on  the  blue  side  of  the  encounter 
are  then  systematically  varied  and  the  effect  on  the  level 
playing  field  recorded.  The  modification  process  is 
repeated  with  the  models  of  the  blue  submarine  held 
constant  and  the  models  of  the  red  submarine  varied  to 
test  the  reciprocity  of  the  effects. 

Initial  implementation  of  the  ASW  models  was  completed 
in  the  Sensible  Agent  Testbed  [1][2].  The  Sensible  Agent 
Testbed  offers  a  highly  composable,  distributable 
simulation  environment.  Modeling  functionality  is 
encapsulated  in  components  called  Effects  Modelers  that 
interact  through  a  message  broker.  Four  different  Effects 
Modelers  were  used  to  model  the  ASW  domain:  (1)  a 
model  of  acoustic  signal  propagation  from  acoustic 
emitters  to  acoustic  sensors,  (2)  a  kinematic  model  of 
submarines  and  torpedoes,  and  (3&4)  models  of  the 
commanders  of  both  submarines.  All  of  the  Effects 
Modelers  are  highly  configurable,  enabling  a  wide  range 
of  experiments.  The  Sensible  Agent  Testbed  also  employs 
a  simple  yet  powerful  mechanism  for  specifying  which 
data  to  collect  during  an  experiment  using  topical  tags 
and/or  component  names. 

The  scenario  of  the  C3  model  set  emphasizes  features 
differently  than  the  physics-based  ASW  scenario  and  thus 
provides  a  degree  of  robustness  to  the  testing  of  the 


semantic  descriptors.  The  emphasis  of  the  C3  scenario  is 
on  decision  making,  the  relevant  results,  and 
communications.  The  scenario  contains  the  fundamental 
features  of  any  C3  simulation.  It  consists  of  a  fighter 
aircraft,  in  contact  with  a  ground  controller,  which  is 
vectored  toward  a  tanker.  The  aircraft  obtains  the  tanker, 
control  is  passed  from  the  ground  controller  to  the  tanker, 
and  the  final  approach  is  executed.  This  scenario  provides 
a  second  means  of  testing  the  relevance  of  semantic 
descriptors.  Instead  of  measuring  the  effect  on  a  level 
playing  field  in  force  on  force  encounters,  for  this 
scenario  a  baseline  is  defined  and  statistical  parameters 
collected.  The  representation  of  the  pilot  and  controller 
are  at  comparable  levels  for  this  baseline.  Excursions  are 
then  conducted  to  examine  the  impact  of  implemented 
changes. 

5.2  Limitations  on  initial  experiments 

Both  of  the  experimentation  scenarios  have  stochastic 
elements  to  them.  Therefore,  to  establish  the  baseline  for 
the  C3  scenario  and  the  definition  of  a  level  playing  field 
for  the  ASW  scenario  requires  a  statistically  significant 
number  of  simulation  runs.  Similarly,  each  instance  of  a 
variation  will  require  an  equal  number  of  runs.  For  the 
ASW  scenario,  seven  aspects  of  the  models  have  been 
identified  for  change  for  the  submarines.  Most  of  these 
could  also  be  applicable  to  the  torpedoes.  For  the  C3 
scenario,  five  aspects  have  been  identified.  These  aspects 
are  listed  in  Table  2  along  with  a  nominal  minimum 
number  of  variations  for  each  aspect. 


Model  aspect 

Nominal  minimum 
variations 

ASW  Scenario 

Update  rate 

5 

Bounded  volume  resolution 

3 

Damage  assessment  detail 

3 

Radiated  noise  resolution 

5 

Movement  dynamics 

3 

Navigation 

3 

Array  gain  resolution 

5 

C3  Scenario 

Degrees  of  freedom 

3 

Decision  representation 

3 

Elapse  time 

3 

Organizational  stratification 

2 

Systems  impacted 

2 

Table  2.  Model  aspects  associated  with  candidate  descriptors 
for  experimental  scenarios 


For  exhaustive  testing  of  all  possible  combinations  or 
variations,  the  total  number  of  simulation  runs  grows 
geometrically  with  the  number  of  aspects  to  be  changed. 
Even  for  the  small  model  sets  in  these  experiments,  the 
total  number  of  trials  required  quickly  becomes 


prohibitive.  In  theory,  exhaustive  testing  would  only  be 
required  if  there  were  no  independence  between  any  of 
the  aspects.  For  aspects  that  are  independent  of  all  the 
others,  their  variations  could  be  rune  while  holding  all 
other  aspects  to  their  values  of  the  base  case.  No 
additional  information  about  the  semantic  descriptor  or 
metrics  associated  with  that  aspects  would  be  gained  from 
additional  runs  made  with  varying  the  values  of  the  other 
aspects.  If  all  the  aspects  for  a  given  model  set  were 
independent  from  each  other,  the  total  number  of  required 
simulation  runs  rises  arithmetically  rather  than 
geometrically.  The  total  number  of  simulation  runs 
required  to  characterize  all  the  semantic  descriptors  and 
associated  metrics  will  typically  lie  between  the 
arithmetic  and  geometric  limits. 

It  was  beyond  the  scope  of  the  current  effort  to  identify 
and  characterize  the  dependence  of  pairs  or  groups  of 
semantic  descriptors.  An  example  of  a  likely  dependent 
pair  from  the  ASW  scenario  is  array  gain  resolution  and 
radiated  noise  resolution.  One  could  ask  at  what  level  of 
resolution  of  array  gain  is  no  significant  difference  in 
detection  rate  noticed  for  a  source  that  is  modeled  as 
omni-directional. 

In  the  experimental  designs  for  this  effort,  the  assumption 
was  made  that  all  the  aspects  investigated  are  independent 
of  each  other.  This  assumption  was  made  for  two  reasons; 
to  keep  the  total  number  of  required  simulation  runs 
manageable,  and  because  the  question  of  semantic 
descriptor  interdependence  was  beyond  the  scope  of  this 
investigation.  Because  of  this  assumption,  the  results  of 
the  experimentation  must  be  considered  as  preliminary. 
They  are  sufficient,  however,  to  provide  a  first  order 
assessment  of  the  semantic  descriptors  and  associated 
metrics. 

5.3  Results 

At  of  the  time  of  this  paper,  only  the  first  few  runs  of  the 
ASW  scenario  needed  to  define  the  level  playing  field 
have  been  completed.  The  most  current  results  on  the 
validity  and  applicability  the  candidate  descriptors  for  this 
scenario  will  be  presented  at  the  Workshop. 

6.  Conclusions  and  future  direction 

This  research  into  semantic  descriptors  has  so  far  resulted 
in  a  working  definition  of  semantic  descriptors  for  models 
and  simulations.  It  has  also  provided  a  recommended 
infrastructure  for  the  organization  and  definition  of  these 
descriptors.  It  has  also  provided  an  approach  for  the 
assignment  of  metrics  to  be  associated  with  each  semantic 
descriptor.  It  has  also  provided  an  initial  prototype  set  of 
semantic  descriptors.  It  has  also  provided  prototype 


experimental  techniques  for  evaluating  the  validity  and 
applicability  of  candidate  descriptors. 

While  this  research  effort  has  made  significant 
accomplishments,  there  is  still  much  to  be  done.  A  few 
specific  follow-on  efforts  are  recommended. 

•  This  effort  has  proposed  eight  different  functional 
domains  for  the  organization  of  semantic 
descriptors.  This  is  not  a  complete  set.  One  follow- 
on  effort  would,  therefore,  be  to  develop  a 
necessary  and  complete  set  of  functional  domains 
and  test  the  completeness  of  this  set. 

•  The  test  scenarios  used  in  this  deal  with  descriptors 
at  the  mid  (or  platform)  level  of  the  hierarchical 
domains.  A  logical  extension  would  be  to  develop 
prototype  descriptors  and  their  associated  metrics 
that  address  models  at  the  other  levels  of 
abstraction. 

•  The  current  effort  developed  a  set  of  semantic 
descriptors  that  are  good  in  the  context  of  the  test 
scenarios.  A  logical  next  step  would  be  to  test  the 
utility  of  these  descriptors  in  the  context  of 
multiple  scenarios.  How  well,  for  example,  can  the 
descriptors  developed  in  the  context  of  the  C3 
scenario  address  C3  requirements  added  to  the 
ASW  scenario.  This  would  essentially  test  the 
descriptors  in  a  proof  of  principle  demonstration  of 
composability  by  constructing  simulations  to  meet 
different  requirements  sets  from  a  collection  of 
described  models 
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